Assertion Method

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 362 of xUnit Test Patterns for the latest information.
How do we make tests self-checking?

Call a utility method to evaluate whether an expected outcome has been achieved.

Sketch Assertion Method embedded from Assertion Method.gif

A key part of writing Fully Automated Tests (see Goals of Test Automation on page X) is to make them Self-Checking Tests (see Goals of Test Automation) to avoid having to inspect the outcome of each test for correctness each time it is run. This involves finding a way to express the expected outcome in a way that can be verified automatically by the test itself.

Assertion Methods give us a way to express the expected outcome in a way that is both executable by the computer and useful to the human reader who can then use Tests as Documentation (see Goals of Test Automation).

How It Works

We encode the expected outcome of the test as a series of assertions that state what should be true for the test to pass. The assertions are realized as calls to Assertion Methods that encapsulate the mechanism that causes the test to fail. The Assertion Methods may be provided by the Test Automation Framework (page X) or by the test automater as Custom Assertions (page X).

Why We Do This

Encoding the expected outcome using Conditional Test Logic (page X) is very verbose and makes tests hard to read and understand. It is also much more likely to lead to Test Code Duplication (page X) and Buggy Tests (page X). Assertion Methods help us avoid all these issues by moving that complexity into reusable Test Utility Methods (page X) that can be verified to work correctly using Test Utility Tests (see Test Utility Method).

Implementation Notes

All the members of the xUnit family provide Assertion Methods but it is an area where there is a fair degree of variability. The key implementation considerations are:

How to call the Assertion Methods,
How to choose the best Assertion Method to call, and
What information to include in the Assertion Message (page X).

Calling Built-in Assertion Methods

The way the Assertion Methods are called from within the Test Method (page X) varies from language to language and framework to framework. The language features determine what is possible and preferable while the framework builders chose which of several options to use. The names they choose for the Assertion Methods are influenced by how they chose to access them. The most common variations for accessing the Assertion Methods are:

The Assertion Methods are inherited from a Testcase Superclass (page X) provided by the framework. This allows them to be invoked as though they are provide locally on the Testcase Class (page X). The original versions Java's JUnit, for example, use this approach by providing a Testcase Superclass that inherits from the calls Assert which contains the actual Assertion Methods.
The Assertion Methods are provided via a globally accessible class or module. They are invoked using the class or module name to fully qualify the Assertion Method name. NUnit, for example, uses this approach. (e.g. Assert.isTrue(x);) JUnit does allow assertions to be invoked as static methods on the Assert class (e.g. Assert.assertTrue(x)) but this is not usually necessary because they are inherited via the Testcase Superclass.
The Assertion Methods are provided as "mix ins" or macros. Ruby's Test::Unit, for example, provides the Assertion Methods in a module Assert that can be included in any class(This is particularly useful when building Mock Objects (page X) because they are outside the Testcase Class but need to invoke Assertion Methods.) thereby allowing the Assertion Methods to be used as though defined within the Testcase Class (e.g. assert_equal(a,b).) CppUnit, on the other hand, defines the Assertion Methods as macros which are expanded before the compiler sees the code.

Assertion Messages

Assertion Methods typically take an optional Assertion Message as a text parameter that is included in the output when the assertion fails. This allows the test automater to explain to the test maintainer exactly which Assertion Method failed and to better explain what should have occurred. The error detected by the test will be much easier to debug if the Assertion Method provides more information about why it failed. Choosing the right Assertion Method goes a long way to achieving this because many of the built in Assertion Methods provide useful diagnostic information about the values of the arguments. This is especially true for Equality Assertions.

One of the biggest difference between members of the xUnit family is where the optional Assertion Message is in the argument list. Most members tack it on to the end as an optional argument but at least one, JUnit makes it the first argument when it is present.

Choosing the Right Assertion

We have two goals for the calls to Assertion Methods in our Test Methods:

Fail the test when something other than the expected outcome occurs, and
document how the system under test (SUT) is supposed to behave. This goal is known as Tests as Documentation.

To achieve these goals we must strive to use the most appropriate Assertion Method. While the syntax and naming conventions vary from one member of the xUnit family to the next, most provide a basic set of assertions that fall into the following categories:

Single Outcome Assertions such as fail; these take no arguments because they always behave the same.
Stated Outcome Assertions such as assertNotNull(anObjectReference) and assertTrue(aBooleanExpression); these evaluate a single argument.
Expected Exception Assertions such as assert_raises( expectedError) { codeToExecute }; these evaluate a block of code and a single expected exception argument.
Equality Assertions such as assertEqual(expected, actual); these compare two objects or values for equality.
Fuzzy Equality Assertion such as assertEqual(expected, actual, tolerance); these determine whether two values are "close enough" to each other by using a "tolerance" or "comparison mask".

We need to verify different kinds of outcomes in a Fully Automated Test so most members of the xUnit family provide several different forms of Assertion Method:

Variation: Equality Assertion

Equality Assertions are the most common examples of Assertion Methods. They are used to compare the actual outcome with an expected outcome that is expressed in the form of a constant Literal Value (page X) or an Expected Object (see State Verification on page X) By convention, the expected value is specified first and the actual value follows it. The diagnostic message output by xUnit typically depends on this order. The equality of the two objects is usually determined by invoking the equals method on the expected object. If the SUT's definition of equals makes this impractical, we can either do Equality Assertions on individual fields of the object or we can use a Test-Specific Subclass (page X) that implements our test-specific equality as the Expected Object.

Variation: Fuzzy Equality Assertion

When we cannot guarantee an exact match due to variations in precision or expected variations in value, it may be appropriate to use a Fuzzy Equality Assertion. Typically, these look just like an Equality Assertion with the addition of an extra "tolerance" or "comparison map" parameter that specifies how close the actual argument must be to the expected one. The most common example of a Fuzzy Equality Assertion is the comparison of floating point numbers where the limitations of arithmetic precision needs to be accounted for by providing a tolerance (the maximum acceptable distance between the two values.)

I have used the same approach when comparing XML documents where direct string comparisons may result in failure due to certain fields having unpredictable content. In this case, the "fuzz" specification is a "comparison schema" that specifies which fields need to match or which ones should be ignored. This particular kind of equality assertion is very similar to asserting that a string conforms to a regular expression or other form of pattern matching.

Variation: Stated Outcome Assertion

Stated Outcome Assertions are a way of saying exactly what the outcome should be without passing an expected value as an argument. The outcome must be common enough to warrant a special Assertion Method. The most common examples of this are:

assertTrue(aBooleanExpression) which fails if the expression evaluates to FALSE and
assertNotNull(anObjectReference) which fails if the objectReference doesn't refer to a valid object.

Stated Outcome Assertions are often used as Guard Assertions (page X) to avoid Conditional Test Logic.

Variation: Expected Exception Assertion

In languages that support block closures, we can use a variation of Stated Outcome Assertion that takes an additional parameter specifying the kind of exception we expect. We can use this Expected Exception Assertion to say "run this block and verify that the following exception is thrown." This is more compact than using a try/catch construct. Typical examples are:

should: [aBlockToExecute] raise: expectedException in Smalltalk's SUnit, or
assert_raises( expectedError) { codeToExecute } in Ruby's Test::Unit.

Variation: Single Outcome Assertion

A Single Outcome Assertion always behaves the same. The most commonly used Single Outcome Assertion is fail which causes a test to be treated as a failure. It is most commonly used in two circumstances:

As an Unfinished Test Assertion (page X) when a test is first identified and implemented as an nearly empty Test Method. By including a call to fail, we can have the Test Runner (page X) remind us that we still have a test to finish writing.
As part of a try/catch (or equivalent) block in an Expected Exception Test (see Test Method) by including a call to fail in the try block right after the call that is expected to throw an exception. If we don't want to assert something about the exception that was caught we can avoid an empty catch block by using Single Outcome Assertion success to document that this is the expected outcome.

One circumstance in which we really should not be using Single Outcome Assertions is in Conditional Test Logic. There is almost never a good reason to include conditional logic in a Test Method as there is usually a more declarative way to do it using other styles of Assertion Methods. For example, use of Guard Assertions results in tests that are more easily understood and less likely to yield incorrect results.

Motivating Example

The following illustrates the kind of code that would be required for each item we wanted to verify if we did not have Assertion Methods. All we really want to do is

      if (x.equals(y)) {
         throw new AssertionFailedError( "expected: <" + x.toString() +
               "> but found: <" + y.toString() + ">");
      } else  { // OK, continue // ...}
Example UnsafeChecking embedded from java/com/xunitpatterns/misc/LifeWithoutAssertions.java

but this will cause a NullPointerException if x is null and it would be hard to distinguish this from an error in the SUT. So we have to put some guard clauses around this so that we always throw an AssertionFailedException:

      if (x == null) { // cannot do null.equals(null)
         if (y == null ) {  // they are both null so equal
            return;
         } else {
            throw new AssertionFailedError(
               "expected null but found: <" + y.toString() +">");
         }
      } else if (!x.equals(y)) { // comparable but not equal !
         throw new AssertionFailedError( "expected: <" + x.toString() +
                  "> but found: <" + y.toString() + ">");
      } // equal
Example BetterChecking embedded from java/com/xunitpatterns/misc/LifeWithoutAssertions.java

Yikes! That got pretty messy. And we'll have to do this for every attribute we want to verify? This is not good. There must be a better way.

Refactoring Notes

Luckily for us, the inventors of xUnit realized this and have already done the requisite Extract Method[Fowler] refactoring to create a library of Assertion Methods that we can call instead. We simply replace the mess of in-line if statements and thrown exceptions with a call to the appropriate Assertion Method. The example below is the code for the JUnit assertEquals method. Note that although the intent is the same as the code we wrote, it has been written in terms of guard clauses that identify when things are equal.

   /**
    * Asserts that two objects are equal. If they are not
    * an AssertionFailedError is thrown with the given message.*/
   static public void assertEquals(String message, Object expected, Object actual) {
      if (expected == null && actual == null)
         return;
      if (expected != null && expected.equals(actual))
         return;
      failNotEquals(message, expected, actual);
   }
Example EqualityAssertionImplementation embedded from java/junit/framework/Assertions.java

The method failNotEquals is a Test Utility Method that fails the test and provides a diagnostic assertion message.

Example: Equality Assertion

Here is the same assertion logic recoded to take advantage of JUnit's Equality Assertion:

      assertEquals( x, y );

Inline code sample

Here is the same assertion coded in C#. Note the classname qualifer and the resulting difference in the method naming:

      Assert.AreEqual( x, y );

Inline code sample

Example: Fuzzy Equality Assertion

To compare two floating point numbers (which are rarely ever really equal, we specify the acceptable different using a Fuzzy Equality Assertion:

      assertEquals( 3.1415, diameter/2/radius, 0.001);
      assertEquals( expectedXml, actualXml, elementsToCompare );
Example FuzzyEqualityAssertion embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

Example: Stated Outcome Assertion

To insist that a particular outcome has occurred, we use a Stated Outcome Assertion such as:

      assertNotNull( a );
      assertTrue( b > c );
      assertNonZero( b );
Example StatedOutcomeAssertion embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

Example: Expected Exception Assertion

This is an example of how we verify the correct exception was raised when we have blocks. In Smalltalk's SUnit, it looks like this:

      self
         should: [Flight new mileage: -1122]
         raise: RuntimeError new 'Should have raised error'
Example ExpectedExceptionAssertionSUnit embedded from Smalltalk/Test Templates.st

The should: indicates the block of code to run (surrounded by square brackets) while the raise: specifies the expected exception object. In Ruby, it looks like this:

      assert_raises( RuntimeError, "Should have raised error")
                  {flight.setMileage(-1122) }
Example ExpectedExceptionAssertionRubyUnitCompact embedded from Ruby/TestTemplates.rb

Ruby also lets us use this more "control structure" style syntax by delimiting the block using do/end instead of curly braces:

   assert_raises( RuntimeError, "Should have raised error") do
      flight.setMileage(-1122)
   end
Example ExpectedExceptionAssertionRubyUnit embedded from Ruby/TestTemplates.rb

Example: Single Outcome Assertion

To fail the test, use the Single Outcome Assertion:

      fail( "Expected an exception" );
      unfinishedTest();
Example SingleOutcomeAssertion embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

Page generated at Wed Feb 09 16:39:45 +1100 2011